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(S) Rendering a color image for an output medium from symbolic image data. 

(57) A method of rendering a colour image on a 
designated output medium (49) is disclosed 
wkiicti maps colors to the gamut (44) of the 
designated output medium (49) performing 
gamut mapping (FIG. 9) eariier in the image 
synthesis process than current gamut method 30 
methods, at the point where infonmation about 
object primitives and their spectral attributes in 
a scene description Is avaOable. but after the 
fixed scene geometry has been determined by 
the rendering system. The method makes use of 
the output of a symbolic rendering system 
which produces symbolic pbcel expressions 
(80), having basis spectra variables which repre- 
sent the interplay of light and object primitives 
in the scene description, and spectral data (84) 
having colour information about the light and 
object primitives in the scene, and which is 
indexed to the basis spectra variables. The 
method perfonns spectral change calculations 
(90) using the symbolic pixel expressions (80), 
the spectral data (84), and spectral infonmation 
about the gamut (44) of the specific output 
medium (49) to determine the modifications (86) 
to the original spectral data that need to be 
made to the individual object primitives in the 
scene in order to produce image pixel colours 
which are in the gamut (44) of the output 
medium (49). The resulting image colours are 
locally and globally consistent with the seman- a&. 
tics of the image, make effective use of as much 
of the available gamut of the display device as is 
possible, and require no further post-rendering 
gamut mapping prior to display or reproduction 
on the designated medium. 
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The present Invention relates generally to the field of computer graphics software and to the rendenng of 
computer-synthesized color images, and more particularly to a method for rendering computer-synthesized 
color images having image colors which are in the color gamutof a designated output medium, and a machine 
for implementing such a method. . • 

Computer graphics software and rendering systems provide for the computer synthesis and reproduction 
of realistic color images and are capable of producing images that include the entire range of visible colors 
The broad functions of the typical, computer-based Image creation process are illustrated in. and descnbed 
with reference to. FIG. 1 3 of USSN 07/965,685 (hereafter "Ref. 1"). a copy of which was filed with the presen 
application. As a scene is constructed by the scene designer, an application-specific model captures data about 
object primitives, and relationships among them, that comprise the scene, in box 20. The scene descnption. 
stored in an application database 22. contains descriptions of primitives that define the shapes of components, 
objects, and light sources in a scene, appearance attributes of the primitives such as color and surface texture 
and connecUvity relationships that show how object primitives are positioned and fit together in the scene. The 
scene description must then be converted into the set of image pixels for display of the image on an output 
medium, typically on a monitor or in hardcopy form. The process of mapping the scene descripbon to an .mage 
pixel file is typically referred to as "rendering", and the implementation of rendering is referred to as the ren- 

derer". shown in FIG. 13 of Ref. 1 by dotted line box 24. ^..„i„„^ 
As can be seen from the functional steps illustrated in FIG. 1 3 of Ref. 1 , any changes that a scene designer 
may want to make to the image that affect the object primitive level of the image, such as the coloure of the 
objects, will mean changing the scene description database 22 and re-rendering the image, which is a com- 
putationally expensive procedure. This includes making changes to the colore o^ objecte in an "nage. since 
information about objects, including their colore, is only available in the scene descnption 22. Metrv^v. 
changes to the image made beyond rendering pipeline 24 must be made at the pixel Manually ch^^^^^^^ 
object or light source colore at the pixel level, while still maintaining the image realism, is difficult for the scene 

designer to accomplish accurately. 

in another computer-based image creation process, a scene designer is permitted to manipulate scene 

description cotore. surface properties, and light intensities without the ~'"P;^!'°"«' «'^«;^.?f "l^)^^^ 
the entire image. This method is described in detail by Carlos H. Sequin and Eliot K. SmyrI in ^^"'^^ 
Ray Tracing-. Proceedings of SIGGRAPH 1989. July 31 - August 4, 1 989 Boston Mass ,n Comp"tBrJ3^ph,cs 
23 (July 1989), pp. 307 - 314. (hereafter. "Sequin"). That method is illustrated in. and decussed in detail with 
reference to. FIG. 1 of Ref. 1 . Briefly, the method modifies a com^enttonal ray tracing program of the type which 

could be used in shaded rendering process 30 of FIG. 1 3 of Ref. 1 . to produce a P?«;"f '^^-^f ' " '^^^^ 
expressionforeach pixel asthesceneisrendered. Rendering system25.nFIG.1of Ref lutdizes^ 

symbolic shaded rendering process to produce a symbolic pbcel expression for each pDcel m the rendered .m- 

^°*'aii of the colore physically producible on a particular color output medium Is called the color •;gamuf ^ 
the mediurn. The color gamut of an existing output medium is typically smaller than the set of aH visible colore 
S a,^ possible in a rendered image. Moreover, the scene description is typically created 'ndependentty of 
the output medium on whteh it is subsequently displayed or reproduced. The algonthms used in shaded ren- 
dering process 30 for modeling the light striking a particular point on an object may very we I generate a value 
for theSS of one of the object's pUels that is not within the range of the valid values n the color gamut of 
he output medium, or that numerically represents a non-visible color, such as a negative value In most in- 
stencSs a "gamut r^apping" process 40 is necessary to make sure that the color attribute for each pixel .n the 
rg?fsdisZableofreproducibleonthedevicesele^ 

40 is described in more detail below. In conjunction with the discusston a""'"^ "/""S/ ^^-l',. 
The technique of producing symbolic pixel expressions illustrated in FIG. 1 ^^^^^^•'•^^^^^'^'^l 
reducing the time involved in changing colore and light intensities in a scene, may P^'^"°«.^'"«' "^^^^^ 
«,lore which are outside the gamut of the output medium which will display or reproduce the image, since the 
"aTe is created independenUy of the output medium on which it is ^^^^^^ 
Thus as shown in FIG 1 of Ref. 1. the pixel colore of image pixel fie 32 most likely wdl need to be mapped 
tothe s^i^rrorgamutofthe output medium,ingamutmapping process 40,beforedispl^ 

GamufmaSng process 40 includes gamut mapping step 42 which applies information abou t he avaMable 
colorgTmut of the output medium from device gamut data file 44 to automated mapping algonthms in orter 
to bring out-S-gamut pixel colore inside the device gamut. TyplcaHy. device gamut data f fle 44 is comprised 
Sdrft^n^rsSesetofactualcolorefromthecolorgamutoftheoutputmedium which have b^^^^ 

latredbyanTppropriate instrument, such asacolorimeterorspeclrop 

S gTp 42 is a r^odifted set of image pixel colore, shown in data file 46, that are now '^^r^^^^l^^^^^^^^ 
Su^ut medium. Since each device gamut is different, a new set of image pixel colore is created by the auto- 



EP 0 594 370 A1 



mated gamut mapping process for each output medium on which the image Is to be displayed or reproduced. 

The goal of gamut mapping is to fit the Image colors into the gamut of the output medium white maintaining 
overall appearance of the Image. The gamut mapping process has generally been the concern of those In- 
volved in the reproduction or display of color Images, and typically has not been a processing goal of the graph- 

5 ics rendering pipeline 24 (FIG. 13 of Ref. 1) or symbolic shaded rendering system 25 (FIG. 1 of Ref. 1), nor a 
primary concern of the scene designer It is a post-rendering step, accomplished typically by using various 
automated methods implemented in software which globally or locally modify Individual pixel colors without 
regard to the objects in the Image from which the pixels are derived. Gamut mapping may also be accomplished 
manually. Typically, the gamut of the output device is mathematically represented as a three-dimensional vol- 

10 ume In a color space, and pixel colors are presented as points in the color space. One common automated 
gamut mapping method is a "uniform global scaling" of all of the pixels in the image such that the pixel colors 
most outside the gamut volume of the device are brought within the gamut region in the color space, and all 
other pixel colors are seated in proportion. Uniform global scaling is conceptually similar to "turning down the 
light" in the scene description. In some, or perhaps many, cases, global scaling of the Image pixel colors dark- 

IS ens and desaturates the resulting Image unacceptably. Another common automated gamut mapping method 
involves the "local correction" of pixels, where each pixel is independently examined and brought into the gamut 
of the specific display device on which the Image is to be produced. One undesirable effect of most methods 
of modifying the out-of-gamut pixels atone, apart from the objects in the scene, is the introduction of shading 
artifacts into the image. 

20 Another highly undesirable effect of all gamut mapping done at the pixel level is to produce an Image which 
Is no longer semantically consistent with the interplay of light with objects in the scene. The concept of image 
semantic inconsistency and problem with the existing process of modifying individual image pixels of objects 
having out-of-gamut colors is discussed In detail in Ref. 1 with reference to FIG. 2 thereof. 

For example, in a synthesized image containing several brightly coloured objects and also containing re- 

25 flections (56 and 58) of these objects in a window (55), generated by rendering pipeline 24 (FIG. 1 3 of Ref. 1), 
the colours of the reflected objects are semantically consistent colours of the "real" objects. If the colours of 
the real objects are not in the gamut of a particular output device, correcting the colours using uniform global 
scaling of the image pixel colors will darken both of the objects as well as their reflections, and darkening of 
the reflections may result in considerable toss of object detail. It Is found that local correction of only the out- 
do of-gamut (red) pixels of a certain object (truck 52) to bring them within the gamut of the device will result in a 
semantically Inconsistent image where the reflections of the object reflect a different relationship between the 
relative brightness of the objects than what Is seen between the objects themselves, once corrected for out- 
of-gamut colors. Alternatively, after the rendering of the image is completed, a user might manually control 
parameters in automatic mapping algorithms In order to adjust, through trial and error and using artistic judg- 

35 ment, the individual out-of-gamut pixel colors for each of the pixels in the image in order to bring the pixel colors 
in-gamut in a manner which achieves or almost achieves semantic consistency. This solution, however, rarely 
provides image semantic consistency because of the complexity of the task, and requires a manual adjustment 
step which typically involves considerable trial and error because of the relatively unpredictable nature of the 
changes involved. Thus, semantic inconsistency resulting from a post-rendering gamut mapping process can 

40 arise in most rendered scenes which model diffuse and specular inter-reflection of light in the scene. 

One method for achieving both in-gamut pixel colors and semantic consistency of object colors and lights 
in the image for the device on which the inrtage is to be displayed or reproduced, is for the scene designer to 
try to make color changes to the objects in the original scene description database in order to bring the out- 
of-gamut colors of certain objects in gamut. If the scene designer is using a symbolic rendering system, such 

45 as the one illustrated In FIG. 1 of Ref. 1, re-rendering the scene would not be necessary, but manual adjust- 
ments to the colors in the scene would still be required. While this solution of changing the scene description 
produces semantically consistent images, it requires the manual adjustment of the scene description or the 
symbolic pixel expressions, which can be a time consuming process, especially if re-rendering is involved. 
Because the spectrum, or color description, for one object or light primitive in scene description 22 (FIG. 

50 2) may appear In more than one symbolic pixel expression 80, a modification to one color description may cor- 
rect some out-of-gamut pixel colors while producing other pixel colors which were formerly in-gamut but be- 
come out-of-gamut as a result of the modification. Thus, the problem of directly modifying the original color 
descriptions to produce an In-gamut Image is typically more complex than simply bringing all of the color de- 
scriptions into the gamut of the selected output device. In addition, any single set of modifications directly to 

65 the original color descriptions could produce an image having all in-gamut pixel colors, but that image may not 
satisfy the image metric. 

What is needed, therefore, is a method of rendering a color image with semantically consistent object col- 
ors that are always within the measured gamut of the specific output medium on which the image is to be re- 
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produced or displayed, thus avoiding both the manual and automatic correction of image pixel colors which 
introduce Image and object semantic Inconsistencies during post-rendering pixel color adjustments. 

The present invention provides a method of rendering a colour Imagefrom symbolic image data, according 
to claim 1 of the appended claims. ^ . 

The method of the present invention, in particular, is a device directed method of performmg the gamut 
mapping process 40 of FIG. 1 of Ref. 1 earlier in the image synthesis and rendering process, at the point where 
information about object primitives and their color attributes in a scene description is available, but after the 
point where the fixed scene geometry has been determined by a conventional rendering method. Preferably, 
the method uses the symbolic color image data and the color description data which are the output of a sym- 
bolic shaded rendering system. The symbolic color image data, in the form of symbolic pixel expressions, di- 
rectly representthe Interplay of light and object primitives in the original scene description . The color descripton 
data contain the light intensities and colors of the object primitives in the scene, which have been separated 
from the scene geometry itself, but which are associated with the symbolic color image data via unique iden- 
tifiers such as indices. The method of the present invention is premised on the discovery that since the light 
and color values have been separated from the scene geometry, they are available for modification with re- 
spect to the color gamut of a particular color reproduction device, and the effect on the original scene of every 
modification made to the the light and color values will be captured in the terms of the symbolic expressions. 
Thus the method also uses device color data which represents the color gamut of the color output medium. 
Using the symbolic color image data, the indexed color description data, and the device gamut date, the meth- 
od performs spectral change calculations to determine modif icaUons to the indexed color descnpUon data tha^ 
produce colors which are in the gamut of the color output medium, and which, when used with the symbolic 
color image data in a later step, will generate a target color image where each target color is one of the device 
colors in the gamut of the color reproduction device. Because the symbolic pixel expressions 8° |naintain p o- 
ture infbrmation in a symbolic form at the object primitive level, any changes to the colour and '9^* 'ntensi V 
values which are substituted in the terms of the expressions result in changes at the object ieve in the colour 
image thus changing not only object colours, but also changing the portions of the image that are affected 
Tuy the Changes in object colours. This occurs because the method 90 of the present Invention simulteneously 
changes all of the appropriate symbolic color terms wherever they occur in the symbolc pixel expressions 80 
All Image pixels which have light contributions from an object primitive with a changed colour value are thus 
changed, and the Image remains semantically consistent while the colours are brought in gamut for the des- 

'°"?h?method of the present invention thereby preserves semantic image consistency in each rendering, 
without the need to re-render the image, by utilizing the symbolic pixel expressions and associated color de- 
scriptlons to represent information about object primitive relationships in the rendered color image, and pro- 
dSmage pLl colors which make effective use of as much of the available gamut of the J'sP'ay ^^v^ «s 
is possible and require no further post-rendering gamut mapping prior to display or reproduction (rf tte.^^^^^ 

in addition, the automated process of producing semantically consistent, in-^amut images c«n ^^V^^ P^ 
formed again to reproduce or display the image another time on the same device, without the need to repeat 
time-consuming manual adjustmente to pixel or scene object colors. . . , . „ 

The method of the present inventton produces a semantically consistent rendered color image and mam- 
teins this image semantic consistency while the final image colors are brought within the gamut ^th« des^J; 
nated output medium, without the computationally expensive step of re-rendenng he ""^S^ " » J« t'^- 
Lsuming post-rendering step of manual pbcel color adjustment Because the symbolic P'^^f ';^P^««^^^^^^^^^ 
malnteln pi<^re information in a symbolic form atthe object primitive level, any changes ^ f «^ J"^ ^jf^ 
intensity values which are substituted in the terms of the expressions result in changes at the object level in 

heSorrage. thus changing not only object colors, but also changing the portions of the image tha are 
b7the Ganges in object colors. This occurs because the method 90 of the present invenbon s.mul- 
te^^ ly changes the ippreprtete symbolic color terms where^^^^^^^ 

pressicns 80. Allimage pixels which havellght contributions from an ot^ctpnmihyewrthachan^^^^^^^^^ 

are thus changed, and the image remains semantically consistent while the a,lore are 'n ga^^^^ f 

the designated output medium. Utaizing the method of device directed rendenng for rendenng the image 50 
cl anZtr sL! scene In FIG. 2 foJa particular output medium would result m bringing the red pixete <rf 
brigSt red fire truck 52 Into the gamut of the output medium, while also mainteining the bnghtness relationship 
between f ire truck 52 and construction cone M in the colors of their reflections in the window 55 

TanotSer aspect of the present invention, there is provided a method of rendering a colour image from 
svmbolic image date according to claim 4 of the appended claims. . ui ^ ^ 

U^pa Llar. and more preferably, the method comprises the steps of assigning "Change vanab^s or 
weights to the symbolic color image date, each change variable having a current value; evaluating the sym- 
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bolic color image data with the indexed colour description data indexed to respective ones of the symbolic col- 
our image data to generate a current color image composed of current colors, the current values for the change 
variables being applied to the indexed colour description data items indexed by the respective symbolic colur 
Image data items; and generating a target color image composed of target colors from the current colors of 

5 the current color image using the device gamut color data representing a gamut of device colours of a colour 
reproduction device; each of the plurality of target colours composing the target colour image being one of the 
device colours in the gamut and corresponding to one of the current colours of the current colour image. Then, 
calculating difference data between the current colors and the target colors is calculated. Using the difference 
data, incremental change quantities for the values of the change variables are determined in order to find the 

10 next set of values for the change variables. A test is then made to determine from the difference data and from 
the incremental change quantities whether the current color image satisfies image metric data. The image met- 
ric data includes relationship data for measuring a valid color Image in terms of a relationship between the cur- 
rent color image and the target color image, and change variable data for measuring a minimum acceptable 
quantity for the incremental change quantities. If the current color image does not satisfy the image metric 

15 data, then the incremental change quantities are applied to update the current values of the change variables, 
and the above steps are repeated again, as long as the image metric is not satisfied. Once the Image metric 
is satisfied, the current values of the change variables assigned to the symbolic color image data are applied 
to the corresponding indexed color description data to produce the modified color description data. 

In accordance with another aspect of the present invention, there is provided a method of operating a ma- 

20 chine for rendering a colour image from symbolic image data, according to claim 6 of the appended claims. 
Preferably, stored in the memory include mathematical operators combining the symbolic spectral components 
into a single symbolic expression. 

Preferably, the step of determining a new value for at least one change variable includes the steps of (b.1) 
determining an incremental change quantity from a current value of the change variable; and b.2) applying 

25 the incremental change quantity to the current value of the change variable to obtain the new value. 

Preferably, each symbolic color Image data Item includes a symbolic pixel function of the change variables 
and the symbolic spectral components; and the step of determining an incremental change quantity includes 
the steps of (i) differentiating each symbolic pixel function with respect to the change variables therein to pro- 
duce differentiated symbolic pixel functions; (ii) constructing a jacobian matrix in the memory of the machine 

30 firom the differentiated symbolic pixel functions; (iii) computing and store in the memory of the machine a pseu- 
do-inverse jacobian matrix from the jacobian matrix; and (iv) evaluating the pseudo-inverse jacobian matrbc 
using the current value of each change variable to find the incremental change quantity for each change va- 
riable. 

Preferably, the desired relationship represented by the image metric data is a distance quantity in a color 
35 space between each of the image pixel color data Items composing the color Image and a corresponding one 
of the target image pixel color data items; and the color image satisfies the relationship represented by the 
image metric data when the distance quantity is minimized. 

Preferably, the method further includes after step (a) and before step (b), the steps of (a.1) initializing the 
change variables with Initial values having no modification effect on the symbolic spectral components; (a.2) 
40 evaluating each symbolic spectral component of each symbolic color image data item with the indexed color 
description data items to produce a plurality of ideal image pixel color data items composing an ideal color im- 
age; and (a.3) storing the ideal image pixel color data items in the memory; and wherein the image metric data 
represent a desired relationship between the plurality of ideal image pixel color data items composing the ideal 
color image and the target image pixel color data items composing the target color Image. 
45 The present invention further provides a programmable image processing device when suitably program- 
med for carrying out the method of any of claims 1 to 7 or any of the above-described particular embodiments. 

In accordance with still another aspect of the present invention, there is provided a machine for rendering - 
a colour image firom symbolic image data, according to claim 8 of the appended claims. 

in summary, the method of the present invention modifies individual object and light source scene spectra 
50 in order to produce a rendered color image having colors that are directly displayable or reproducible on a se- 
lected output medium without an additional gamut mapping step applied to individual image pixels. The Image 
produced by the method preserves semantic relationships and consistency among the objects in the image 
while taking advantage of the widest range of colors available in the output medium's gamut. By changing the 
scene description rather than individual image pixels, the introduction of shading artifacts is avoided, and 
55 changes in the relative brightness of objects and in pixel chromaticity, in certain instances, are permitted. 

Other aspects of the present invention will become apparent as the following description proceeds and 
upon reference to the drawings, in which: 

Figure 1 is a blocic diagram illustrating the data flow for color image rendering using the method of device 
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directed rendering according to the present invention; 

Figure 2 Is a block diagram illustrating tlie functional steps for the method of device directed rendenng 

shownasblocl<90inFIG. 1; i:„„„.^»ho 

Figures 3 and 4 diagrammatically illustrates using w^eighted vectors according to one embodiment of the 
present invention to represent colors in a color space, and modifying the weights to change the colors; 
Figures 5. 6. 7 and 8 conceptually Illustrate a model of an image, or picture, function used according to an 

embodiment of the method of the present invention; 

Figure 9 is a flow chart illustrating the steps of one implementation of the method of the present invention. 
Figures 10. 11 and 12 are block diagrams of suitable machine configurations which may be operated ac 
cording to the present invention; 

Figure 13 illustrates schematically representative data structures used by the present invention; 
In the Figures, the same numbers have been used to denote the same elements. 

In FIG 1 the device directed method for rendering a color Image, illustrated as spectral change calculator 
90 uses as Input the original color and light intensity values for the scene description In indexed color dwcnp- 
tion data 84 and symbdio color Image data, called symbolic pixel expressions 80. both generated by a rendenng 
system, and device gamut color data 44. defining the colors capable of reproduction by <»lor device 49 The 
method performs spectral change calculations to produce as Its output, a set of modified color d^criptions 
86 which are modified color and light intensity values for the scene description. When the modif ied color de- 
scriptions 86 are used, along with the symbolic pixel expressions 80. in the symbolic pixel expression evaluator 
82 image pixel color data 46 composing a color Image is produced which has pixel colore which are substan- 
tially all In the device gamut of device 49. The modified color description data 86. together with he symbolic 
pixel expressions 80. now represent the color Image rendered for device 49. and no gamut mapping step pnor 
to display or reproduction Is needed to ensure displayable colore on output medium 49. It is not necessary to 
perform evaluation step 82. however, until a user or viewer of the color image is ready to display or reproduce 

the image. ^^^^^ ^ performing the spectral change calculations. Each of the originally 

defined color and light intensity valuesfor the scene description 22 In indexed color descilpUon data 84 >smod^ 
fled in box 94. The details describing one way in which the color descriptions are modified is setforth below 
with respect to the illustrated embodiment Generally, the color descriptions must be modified in a manner 
which when all of the modif ied color descriptions are evaluated In the symbolic pixel expressions 80 to produce 
a color image for display, in symbolic pixel expression evaluator 82. the colorlmage which B produced by sym- 
bolic pixel expression evaluator 82 is composed of pixel colore in the device gamut 44 erf device 49. Typify 
the cdor descriptions will be modified in a manner which results in the modified color being in device gamut 
44 although they may be adjusted to meet other similar criteria. The modifications need not be directly made 
to Ihe valfes of the original colore in indexed color description date 84. Since the original colore are indexed 
to and represented in the symbolic pixel expressions, manipulations of the symbolic color terms th^V^Jollc 
pb(el expressions, which result in modifying the values of the color descriptions when they are substituted in 
the symbolic pixel expresstons. may also accomplish the necessary modifications indirectly. 

Once a first set of modifications to the Indexed color description date 84 has been made, he modified 
color descriptions are evaluated, In box 82. In the symbolic pbcel expressions 80 to produce a 

in Sx Too. each image pixel color in the color Image is mapped to the device gamut of a speaf « disptey 
or reproduction device (shown as device 49 In FIG. 1) using a conventional gamut mapping process. TypicaHy. 
the device gamut of a device is conceptually and mathematically represented as «'hree-dimensional volume 
in color space, and a mapping process finds a deslrabte replacement color in the device Samut for ^e M 
gamut cdor. according to a mathematical algorithm for finding a desirable replacement The device gamut 
KveTmiy also be represented in other ways, such as. for example, as simply a look-up table <rf measured 
Sv!^^ for valid device colore. Device gamut data 44 include the date needed to represent the gamut n 
the manner selected by the implementetion. The act of mapping each pbcel color In the current color image to 
JJeScegLt using devicegamutdate44generatesatergetcolorlmag^ 

''Tirintuiry in box 96 teste whether the terget image generated In box 1 00 satisfies an image metric which 
measures wheLr the terget Image is a valid teiget image. The image meWc in Its simplest ^rm "lay s'mply 
^uire that ail evaluated symbolic pixel expressions 80 produce in-gamut pixel colore. However there may 
S^^ny modllcatlons to thVoriginal color descriptions in indexed color description date 84 which would pro- 
one or Ire color images having substentially all pixel colors In the device gamut '«P«»«'««'' ^y t!?,^ 
g^ut date 44. The image metric may determine which one or ones of the possible target oo\orja^^9^^^ 
S terget color Images The image metric may also provide for modifications to the color descriptions to stop 
«^ hin a c^ tl number of processing cycles (in which case any remaining out-of-gamut pixel colore will remain 
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out-of-gamut). 

The determination of validity of a target color image may be established in a number of ways, and a specific 
image metric Is described In more detail below with respect to the illustrated embodiment. Generally, the goal 
of the device directed rendering method of the present invention Is to find a set of color values for the colors 

5 of the objects and lights in the rendered image which are in the device gamut represented by device gamut 
data 44. and which produce an image which is acceptable, for some measurement of "acceptable". Thus, the 
validity measurement of the image metric is typically a measurement of target color image acceptability to the 
user of the method. In one embodiment of the method, the acceptability measurement of the target color Image 
is stated in terms of, or as a function of, the original color Image rendered by the rendering system. An original 

10 color image, also called an "ideal" color image, may be generated by evaluating (using symbolic pixel expres- 
sion evaluator 82) the symbolic pixel expressions with the original indexed color descriptions 84, prior to mod- 
ification, and each target color image generated may be compared in some manner to the ideal color Image 
to determine validity as a measure of whether the target color image is an acceptable substitute for the Ideal 
color Image. The image metric may also, for example, or provide a measurement of the perceptual suitability 

15 of an arbitrary image as a substitute for the original, ideal image. 

If the image metric has been satisfied, the indexed color descriptions as currently modified are the color 
descriptions to be saved as in-gamut color description data 86 (FIG. 1). The modified. In-gamut color descrip- 
tions 86 are the set of color descriptions which produce an in-gamut target picture which satisfies the meas- 
urement expressed in the image metric, and which, together with the symbolic pixel expressions 80, produce 

20 the valid in-gamut target picture. 

If, however, the modified color descriptions do not produce the in-gamut target picture which satisfies the 
image metric, the process flow returns to repeat the modification, evaluation, and mapping steps. It may be 
possible, in a particular implementation of the method steps illustrated In FIG. 2, to simultaneously modify all 
of the indexed color descriptions that require modification to produce the target color image which satisfies 

26 the Image metric. In that Implementation, the image metric would be satisfied with the first target color image 
produced from modified color descriptions, and the method would terminate without repeating steps 94. 82. 
and 100. 

The method of the present invention begins with the output generated by a symbolic rendering system: 
the symbolic pixel expressions 80 and the indexed color descriptions 84. The creation of the scene and the 

30 scene description database are accomplished In the same manner as described In Ref. 1 with reference to 
FIGS. 12 and 1 thereof, utilizing known modeling, painting, or other software tools for creating Images. The 
scene description contains descriptions of primitives that define the shapes of components, objects, and light 
sources in an image, appearance attributes of the primitives such as transparency, color, and surface texture, 
and connectivity relationships that show how objects are positioned and fit together in the image. In the illu- 

35 strated embodiment, each color of an object surface or illumlnant In the scene description that is the input to 
the symbolic rendering system Is represented as a single spectral function of wavelength. For an illuminant, 
the spectrum represents the amount of power at each wavelength; for a surface, the spectrum represents the 
amount of Incident light reflected at each wavelength. However, the method of the present invention is intended 
to also cover other illumination and reflection models. 

40 In the illustrated embodiment, the scene description Is provided to a symbolic rendering system which uses 
a modified ray tracer to produce an image, or picture, in symbolic form having one symbolic pbcel expression 
80 for each image pixel. The ray tracing method assumed one ray per pixel. The method of the present inven- 
tion, however, Is suitable for use with other rendering models, shading equations, and screen sampling pat- 
terns. 

.45 An indexed is assigned to each unique color in the scene, from both illuminants and surface reflections. 
The indices along with their associated color descriptions (spectra) are output, either as a separate file of in- 
dexed color descriptions 84 as shown, ores part of the file of symbolic pixel expressions 80. FIG. 13 illustrates 
a representative structure for the file 84 of data items comprising the Indexed color descriptions. The number 
of indexed color description data items depends on the number of unique color and light intensity values that 

50 exist in the scene description. A single Indexed color description data item 262 is composed of an index field 
264 and a color, or spectrum, data field 266 containing color Information. In illustrated data field 266, the color 
information is represented as RGB Information, but any data format suitable for representing and encoding 
color information may be used. In the implemented embodiment, the color information is represented as an 
array of floating point numbers defining a color as a set of spectral samples. 

55 Each symbolic pbcel expression represents all of the color contributions from objects and lights in the scene 
at that pixel, or, more precisely, the sum of the combined diffuse and specular reflections of the light propa- 
gated from the surface of an object to the observer's eye. To produce the symbolic pixel expressions 80. the 
symbolic rendering system multiplies together the spectra of the scene surfaces and the lights to form the 
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components, or terms, of the symbolic expressions 80, called "basis spectra", which in turn are added to pro- 
duce the color at each pixel. Each basis spectrum component term in a single expression also has an asso- 
ciated constant that depends on the geometry of the intersection and properties of the surface. Specif Ically. 
the expressions are made up of real numbers, arithmetic operators, parentheses, and color symbols (spectral 
references), as described in Sequin. For example, a color with the Index of 3 may be referenced in a symbolic 
pbcel expression simply as "83". The parameters in each expression, when evaluated with actual color values, 
such as those provided in the original scene description 22, together define the spectral power density (SPD) 

of incident light at an image pixel. ^ j 

An example of how a symbolic pixel expression of component basis spectra terms is constructed by the 
symbolic rendering system aiustrates this process more clearly. Assume that a scene description 22 consists 
of one llluminant with a spectrum designated as S,{X), and one surface with spectral reflectance function S2(X.). 
where X denotes wavelength. E(X,) denotes the spectrum of a pixel that receives a ray that carries light from 
the llluminant and is reflected from a plastic surface into the pixel. The simple shading model of the Tenderer 

produces a relationship of the form: 

£(X.) = CiSi(X) S2(X.) + CjSiCX) (1) 
where the c, and Cj are the scalar constants which are functions of the relative geometry of the iiluminants 
and surfaces and the physical properties of thesurfaces. Note that scalars c, and are importantfor rendering 
the final image and are shown for completeness, but are not important to the method of the present invention. 
The two terms on the right hand side of Equation (1) correspond to the diffuse and specular components of 
the light reflected from the plastic surface provided as the example. In more complex scene instances, addi- 
tional "bounces' of a ray through the scene (l.e., the levels in the ray tree constructed by the ray tracer) results 
In additional spectra being multiplied together, resulting In turn in a high-order polynomial In the spectra Actual 
experience with the implementation of the present illustrated embodiment shows that the ray tree usually does 
not exceed two or three levels, which prevents the order of the polynomials at the pixels from rising too high. 

FIG. 13 illustrates a representative data structure for symbolic pixel expression file 80. Asingle symbolic 
pixel expression 252, representing the color information for a single pixel In the color image, is composed of 
basis spectra terms 254 and arithmetic operators 256. Each basis spectra term 254 Includes one or more data 
areas 258 for data produced by the symbolic rendering system not directly related to spectral data, such as 
the scalare Ci and Cj discussed above, and one or more symbolic spectral components 260 which are, in effect, 
indices Into indexed color description data items 84. The use of an Indexing scheme to create indirect refer- 
encing between a symbolic pixel expression 252 and color description data 84 is an efficient implementation 
over using actual color description data in the symbolic pixel expressions, and as noted earlier, any suitable 
indirect referencing arrangement may be used for associating the actual color and light intensity values in color 
description data 84 to the corresponding symbolic spectral components In the symbolic pixel expressions. The 
symbolic spectral components 260 are used for Indexing Into the indexed color desaiption data 84, as shown 
by arrow 268. to retrieve the corresponding color information data, or spectrum data, needed to evaluate the 
symbolic pbcel expression to produce the image pixel color. ... , 

The basis spectra terms in each symbolic pixel expression are preferably converted to tnstimulus values 
describing vectore In tristlmulus space using a "trisUmulus operator", T, such that TIE(X)] = x. where x is the 
three-vector containing the tristimulus values XYZ corresponding to E{X}. If T [Si(A.)S2(X.)l = Xu and T [SiWl 
= X,. then applying the tristimulus operator to both sides of Equation (1 ) yields a linear combination of tristinriulus 
vectors: 

71B(X)] = 7IcSi(X) SaW + c^S,(>.)] (2) 
x= cr[S,(X)S2(X)] + C471(Si(X)] (3) 

= CiXi2 + CjXi (4) 

When this conversion is performed is an implementation decision, and the basis spectra terms may be 
comrerted to tristimulus vector terms by the symbolic rendering system, after the ray tracing step, as in the 
illustrated embodiment, or later, as an Initial process in device directed rendering step 90. The final color of 
each pixel in the rendered image is thus defined. In the symbolic pixel expression, as the sum of one or more 
of the tristimulus vector terms. In the illustrated embodiment, then, color representation and manipulation is 
performed in the tristimulus color space model. Color representation and manipulation may also be earned 
out In other color spaces, since the tristlmulus vectors may be converted further into other color models such 
as CIE color space models, and the RGB or HSV color models, through mathematical convereions well known 
in the art. Additional processing considerations must be taken Into account, however, if a color space is used 
which is a nonlinear transformation away from the tristimulus color space model. . ^ 

The Ilustrated embodiment of the present invention is based on mathematical models shown in FIGS. 3 
to 8. The implementation of the mathematical models is handled as an optimization problem. Determining mod- 
ifications to the Original color descriptions m an efficient manner requires finding a set of optimal modif icauons 
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to make to one or more of the color descriptions simultaneously which both produce an in-gamut, or target, 
color Image and which Is a valid target picture as defined by the Image metric. Since, when suitable color de- 
scriptions are substituted for the symbols in the basis spectra terms, the symbolic pixel expressions ultimately 
determine alt of the image pixel colors, the modifications to the original color descriptions can be efficiently 

5 determined Indirectly, by manipulating the symbols representing the original color descriptions in the basis 
spectra terms in such a manner as to be able to apply the manipulations to the original color descriptions to 
determine the modified color descriptions. 

The mathematical model of the Illustrated embodiment, to be described in detail below, Is summarized as 
follows. The model defines an original rendered image, also known as the idea! image, as a "picture" repre- 

10 sented in "picture space". The picture is defined as a function of the basis spectra (color descriptions 84). Each 
spectrum may be scaled by a weight to adjust its color. Because the spectra are physical representations of 
colors, the weights must meet certain criteria, and so, of all weig hts In "weight space", there is subset of weights 
which are valid. The subset of valid weights when applied to the spectra produce a subset of valid pictures In 
picture space. Because a picture is a function of the spectra, and each spectrum is a function of one or more 

15 weights, a valid picture may be expressed as a continuous function of the weights, and the subset of valid pic- 
tures define a subset In picture space. A device gamut also defines a subset in picture space. A"taiget picture" 
Is defined as a picture which has Image pixel colors that are reproducible colors in the device gamut. The target 
picture may be found by applying a gamut mapping function to the original ideal image. There is some set of 
weights which meet the following constraints: the set of weights produces a valid picture in picture space; the 

20 picture produced is also in the device gamut of picture space; and the picture produced meets some objective 
function, called the Image metric or "goodness function" which describes the desired relationship between the 
original picture In picture space and a target picture In the gamut subspace of picture space. Optimization of 
this mathematical model attempts to find the optimal set of values for the weights that are needed to meet 
these constraints. The details of this model follow. 

25 As noted above, In each symbolic pixel expression, tristimulus vector terms represent individual color con- 
tributions from object and light primitives in the scene to the final pbcel color represented by the expressbn. 
The tristimulus vector terms, in turn, were converted from basis spectra, and so represent the scene spectra 
(objects and lights) which affect the color in the particular pixel represented by the expression. Typically, the 
basis spectra occur as terms in multiple symbolic pixel expressions, because the lights and the objects in the 

30 scene Influence the color In many Individual pixels. 

Figures 3 and 4 illustrate the concept of basis spectra as vectors in the color model known as tristimulus 
space. Note that FIGS. 3 and 4 Illustrate a two-dimensional example, simplified from the three-dimensional 
model Implemented In the Illustrated embodiment, and that the vectors shown demonstrate only one of the 
possible symbolic expressions for a pixel. In FIG. 3, dotted line box 120 represents the volume of tristimulus 

35 space. Within tristimulus space 120, a subset 122 of tristimulus space represents the gamut of a particular 
output medium, also denoted"G" In FIG. 3. The color of a light source, L, In the Image to be rendered is repre- 
sented by the vector in tristimulus space 120 denoted Sl. Similarly, the color of a surface, Ss, illuminated by 
light source, L, in the image is represented by the product of 85 and Sl, denoted SsSf A pixel 1 24 in the image 
has a color which is determined by the sum of SsSl and Sl. Pixel 124 happens to be outside of gamut 122. 

40 For the case where the color of any pixel in the rendered Image is out of gamut, the individual scene spec- 
tra, S, for either the object surfaces (Ss) or the light Intensities (Sl), or both, in the symbolic pixel expression 
which generated the out-of-gamut pixel, are adjusted, or scaled, by a weight, w, in order to bring the vectors 
representing the scene spectra closer to the volume bounded by gamut G. In an Inltlat processing step of device 
directed rendering method 90, each spectral function S|(X,) (an original color description symbol or index) in 

45 Equation (1) is replaced with W|S|(X), a weighted color description symbol or index. Applying the tristimulus op- 
erator T to both sides of Equation (1), as done above in Equations (2), (3), and (4), yields; 

X = (CiX,2)WiVK2 + (C2X,)Wi (5) 

Because x is a tristimulus vector. Equation (5) defines three equations, each a polynomial In the weights. 
Figure 4 Illustrates how individual weights applied to scene spectra bring individual pixels into the gamut 

50 of the output medium. Pixel 124 and gamut 122 are shown in the same positions they have in tristimulus space 
120 of FIG. 3. In the illustrated embodiment, a pixel defined by a single vector can only be brought into gamut 
if the line along the vector Intersects the device gamut. Thus, if the pixel color at point 1 24 In tristimulus space 
were defined only by a single vector along line 124a, a single weight applied to the vector would not be able 
to bring the pixel color into gamut 122, since it would scale the color only along line 124a. However, since pixel 

55 124 in FIG. 3 is defined as a sum of vectors, the pixel color can be corrected (i.e., brought into gamut 122) by 
scaling the weights of the light source and surface spectra as long as the plane or volume bounded by the 
vectors intersects the device gamut. In FIG. 3, In essence, vector SsSl has been scaled with weights having 
values of one (1). In FIG. 4, two corrections are shown. The basis spectrum of light source, L, has been scaled 
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by weight wL and the product of the spectrum and the weight results in a new vector in tristlmulus space 120, 
denoted WlSl- Weight, Wl, has also been applied to the spectrum of light source L In basis spectrum (product 
vector) SsSl Scaling the spectra with weight wl will change the pixel value In a manner defined by the rules 
of vector algebra. Thus, the new product vector in tristlmulus space 120 denoted WlSsSl represents a pixel 
color that has changed from position 124 to position 126 In tristlmulus space. Because only a single weight 
has been applied, the scaling results In pixel color 124 changing along line 124a to point 126, toward the origin 
point of gamut 122. In effect, such a change only results In changing the brightness of the pixel color, and not 
its chromaticity. Pixel color 126 is closer to the boundary of gamut 122, although It Is still an out of gamut color. 
Applying weight Ws and scaling the spectrum of surface Ss results in a new product vector In tristamulus space 
120. denoted WsWlSsSl, representing a new pixel color 128 in tristimulus space which is now within the bound- 
ary of gamut 122. Pixel color 128 is now capable of reproduction in the selected output medium represented 
by gamut 122. 

In the weight system of the Illustrated embodiment, there is a single weight for each spectral function, and 
each weight must meet certain constraints. If the total number of weights is M, each weight is mathematically 
represented as a component of MLdimenslonal vector, w. A particular set of weights defines a point in an M- 
dimensional Euclidian "weight space". Because the spectral functions have a physical interpretation, they can- 
not be scaled arbitrarily. The weight that scales a spectral function must be chosen so that the scaled spectral 
function is physically realizable. In general, weights are positive and nonzero. Surface reflectance functions 
cannot have values greater than 1 .0. W denotes the subset of M- dimensional weight space containing all pos- 
sible w's that produce physically realizable scaled spectral functions. 

Figure 1 3 illustrates a representative data structure for a single symbolic pbcel expression 270 scaled with 
weights. A single unique weight 272 is assigned to each symbolic spectral component 260 In expression 270. 
As a weight takes on values, it functions as a multiplier on the symbolic spectral component 260 to which it Is 
assigned. In turn, the multiplier is applied to the actual color information 266 in indexed color description data 
84 when symbolic spectral component 260 is used to Index Into Indexed color description data 84 during eval- 
uation of expression 270. Ki' -I 

Figures 5 to 8 conceptually illustrate the mathematical model of the problem statement. The N image pixel 
colors which result from evaluating the symbolic pixel expressions 80 each have three tristimulus values per 
pixel and are collectively designated as a "picture" in "picture space". The N image pixel colors of the picture 
are represented by vector p. Vector p is a long vector comprised of 3N tristlmulus values. buOt from a picture 
in scan-line order, and p, denotes the 1*^ component of p, corresponding to one tristimulus value at one pbcel. 
The set of image pixel colors evaluated from the symbolic pixel expressions 80 with the current object and 
light intensity values is designated as the picture. Po. Since the spectra are functions of the weights, and the 
picture Is a function of the spectra, a picture function, f, is defined which maps weights to tristimulus values in 
pixels. The picture function is denoted as p = f(w). P Is the subset of "picture space" containing all pictures for 
which there exists a weightvector w in weight space W such that p = f(w). The picture function Is vector-valued, 
and f, denotes the l^^ component function of p. Each component function is a low-order polynomial function in 
the weights, as illustrated by Equation (5), and thus is continuous and has easily computable denvatives of all 

^'^^Tlinear approximation to the picture function is developed which Is used to search for the optimal set of 
weights to bring the picture into gamut. The matrix of first partial derivatives of the picture function. /Iw), is a 
Jacobian matrix having dimensions of 3N x M. where the entry on the 1*^ row and ^ column is the partial der- 
ivative of fu with respect to Wj (the jth component of w). This Jacobian matrbc Is denoted Jf, Because f is non- 
linear. will depend on w. . *x « 

The linear approximation to p = f(w) is formed by truncating the Taylor senes expansion of f after the sec- 

ond term: 

p + Ap = f(w + Aw) (6) 
p + Ap » f(Wi + J*Aw (7) 
Because p = f(w), the approximation that is made is Ap « J/Aw. 

The linear approximation just described is used to find the changes In the vector of weights. Wo. of the 
current picture function, Po. that are needed to satisfy the picture function that represents the target^p^cture 
p* Targetpicturep*isalsoafunclionofasetofweights,designatedastargetweightsw*,suchthatp - f{vi ) 

The target picture p* is produced by mapping the ideal or current picture to the gamut of the device (described 
in more detail below), so p* is not necessarily in P. since there may be no such that p* = f(w*)^ In the im- 
plemented embodiment, the target weights w* are defined to be the weights needed to minimize some measure 
of distance between p* and the picture function f(wo). The weights, wo. are already Icnown. since po = flwo). 
Thus, the linear approximation of the picture function is used to find the changes in the weights needed to min- 
imize some measure of distance between p* and the picture function f(wo). This distance is defined as a dis- 
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tance function, d(p*, f(w)), and is the image nr^etric hereafter referenced as the objective or goodness function, 
of the optimization problem. 

FIG. 5 illustrates the subset P of picture space, a target picture, p* which is not in P, and the picture in F, 
designated by picture function /(w*), which is the closest, In Euclidian distance, to the target picture. In the 
5 implemented embodiment, the distance function which produced acceptable picture results is defined as: 

d(p\f{w)) = lb* - fM¥ (8) 
where || • || denotes Euclidian distance. Thus, the distance between two pictures is measured by the squared 
Euclidian distance between them in picture space. Other goodness functions are also possible, and may be 
preferable to the distance function of Equation (8) for certain images or certain device gamuts. For example, 

10 a goodness function which describes the minimum distance from p to the most perceptually acceptable sub- 
stitute for p* in G is a desirable goodness function. 

The gamut of the designated output medium is the region of three-dimensional tristimulus space, denoted 
G3, that the device can render. A picture, p, is in gamut when all of Its pixels are In gamut. All In-gamut pictures 
form a subset, G, of picture space P. There exists a "gamut projector" function, h, which maps a picture po to 

15 the picture in G "closest" in Euclidian distance to po- Figure 7 illustrates picture po and the picture. /7(po) in G 
"closest" in Euclidian distance to po Some picture p is the picture that minimizes d(po, p) over alt p in G. Note 
that gamut projector function h is the identity operator when Po is already in 6;/i(pal = p for Po € G. 

Figure 6C illustrates one solution to the stated problem, in the search for the changes to the weights, w, 
for the picture that minimizes d(po, p) over alt p in G, first the gamut projector function h is applied to the picture 

20 Po to obtain the target picture p* = /i(Po). By the definition of h, p* is in gamut, but it may not be in P. Then, a 
nonlinear minimization process (described in detail below) is used to find changes In weights, w, which move 
picture po in P toward p*. The goal of these two steps is to end at f(w*) which defines the picture in G which 
is the smallest distance from p*. Since p* is fixed, the distance that must be found depends only on w. In effect, 
the problem is to minimize a function g(w) - d(p*, f(w)) with respect to w. The function is nonlinear, and a variety 

25 of nonlinear optimization techniques are applicable to the problem. 

The problem of finding the optimum set of weights with which to scale the spectra to produce an in-gamut 
picture can now be briefly restated in the following form. Given an initial picture po. and an initial set of weights. 
Wo, such that po = f(w), the problem is to find the p that minimizes the distance d(po, p) over all p € P n G. 
For a picture p to be in the set P n G, p must be in gamut, and there must be a w € W such that p = ^w). 

30 The specific steps of the Implemented nonlinear optimization technique are as follows. When the distance 
d(p*, f(w*)) is not expected to be too large at its minimum, the Gauss-Newton method may be used to find w*. 
The method is discussed by P. Gill, W. Murray, and M. Wright in Practical Optimization, at pages 134 - 136. 
The method produces a sequence of w's which converge to w*. At each iteration, the next w is computed by 
choosing a direction in which to step w and a length for the step. The Jacoblan of /, Jf, Is used to compute the 

35 step directions, essentially linearizing the problem around the current value of w. Table 1 shows the general 
steps of the implemented method. 



TABLE 1 



40 


Steps in Gauss-Newton IVIethod 




1. 


Evaluate Jf at W|. and let Api = p* - pi; 




2. 


Choose step direction AW| to minimize ||Api - JA^\ II as follows: 


45 




Awj - MAp\ where Jf*- is the pseudo inverse of 




3. 


Choose a step length, k, according to an estimate of how nonlinear f is; 




4. 


Let W| ♦ 1 ^ W| + kAW|; let Pi > 1 «<- (/(w, + 1);. 


50 


5. 


• 

If the sequence {p|} has converged, exit; else let i 4- 1 + 1 and return to step 1. 



Figure 8 illustrates that the steps just described alone will not guarantee that the picture which is the small- 
est distance from p* will be In G. Using the two steps just described results in picture Pi being the closest picture 
to h(po), but Pi is not in G. A third step guarantees that the picture which is the smallest distance from p* will 
55 be in G. At every step in the nonlinear minimization process used to find weights, w, which move picture Po in 
P toward p*. the target picture p* is also recomputed so that there is a sequence of target pictures, p * = /?(pi), 
as shown in FIG. 8. As long as there is some way to move the weights, w, the successive pictures, P|, in P will 
continue to move in P toward the recomputed target pictures, p*. into G. ending at pi as shown. 
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It can be seen from the process just described and from its illustration in FIG. 8. tliat the process imple- 
mented in the Illustrated embodiment does not necessarily keep the recomputed target picture p*, close to the 
originally computed target picture p-p in G. The final p«, will not in general equal the Initial one. The strategy 
just desCTibed treats all p in P r. G as equally good final in-gamut pictures, without regard to their distance 

s from the original image. There may. however, be a modification to the model as implemented which would have 
as Ks goal finding the p in P n G which is the "closest" to the original image, po- 

Figure 9 illustrates the steps In the implemented embodiment of device directed rendering method 90 which 
uses the mathematical model of the picture function to produce a semantically consistent color Image with in- 
gamut pixel colors. First, in box 130, each scene spectrum in each of the symbolic pixel expressions 80 Is as- 

10 signed a variable representing the value for the weight on that spectrum, producing for each pixel expression 
an equation in the form of Equation (5) expressing the pixel color value as a function of the weighte. The weights 
are also Initialized from the input spectra. The Input spectra are read from a vector n. and the values for the 
weights are stored In a vector w. The resulting scaled spectra are stored in vector s. If the Input spectrum, 
n,(3L) is an illuminant. the value of the corresponding weight w, is initially set to one (1), and the spectrum s, 

IS <- n,. The values for input spectra which are yiuminante must be positive, and may be greater than one (1). 
Otherwise, the input spectrum, nKX) is a reflectance. The value for a spectrum which is a reflectance must 
fall between zero and one. and so the value of the corresponding weight w, is Initially set to the largest input 
spectra value, and the spectrum s, n, / w,. Thus, the largest value a weight may take on for a reflectance 
specfrum is one (1). 

20 Other initialization functions may be performed in box 130 as well. For example, a number of user-specified 
thresholds are used to Control subsequent processing steps, and these may be requested interactively from 
the user or otherwise entered into memory at this point 

In box 132. the symbolic pixel expressions 80 are then evaluated to produce a vector of evaluated pixels 
represented as tristimulus values. Evaluation step 132 is substantively similar to evaluation step 82 in FIG. 2. 
25 with modificattons for handling the evaluated pixel expresstons as a vector of spectral cotor information. The 
entire vector of values at this point is the initial picture, p,. also called the "Weal" Image, as produced by the 
symbolic rendering system. All of the color and light intensity values in each symbolic pixel expression 80 which 
generate out-of-gamutpbcel colors may be modified simultaneously, in the processing which foiiovre. Typically, 
processing time and other implementation dependent decisions affect whether to modify ail out-of-gamut col- 
30 ore at one time. In the Illustrated embodiment, a selection of the "worsr pbcel colore is made from all of the 
out-of-gamut pixels to reduce computational time. When these woret pixels are brought into gamut by adjusting 
the weighte on the spectra, the picture is evaluated again, and a new set of "woraP pixels is selected for proc- 
essing The outer loop confrols the selection of the worst out-of-gamut pixels, and an inner loop minimizes the 
picture function to bring the pixels in-gamut However, all out-ol^gamut pixels could be adjusted simultaneously. 
35 if computational time and system resources permit 

The number, b. of "woret" pixel colore selected is arbitrary, and Is primarily influenced by performance con- 
strainte. The manner for selecting the pixels, in boxes 134 and 136, is as follows. Each evaluated pixel color 
is mapped to the device gamut to determine (a) whether the pixel is out-of-gamut; and (b) for each pixel that 
is out-of-gamut (i) what the closest in-gamut pbcel color is. and (ii) what the distance is in the color space 
40 between the out-of-gamut color and the closest In-gamut color. The list of out-of-gamut pixels is sorted in de- 
scending order by distance, and the b number of pixels having the largest distance values are selected for proc- 
essing. The symbolic pixel expressions 80 with weighted spectra which represent these worst pixels form the 
basis of the picture function, Po. which is the current Ideal image. , . ... 

In box 1 38. the inquiry is made whether all. or substantially all. of the pbcels just evaluated are in the gamut 
45 of the designated output medium. This inquiry controls the outer loop of processing. When all pixels are in- 
qamut processing is concluded, and the current set of evaluated pixels colore represent a color image which 
^ semantically consistent and which has image pixels that are within the gamut of the output medium. Contra 
then fransfere to box 154. where the file of modified indexed cotor description data is produced. The current 
values for the weights contain the scate factora to be applied to the original indexed color descriptfons (file 84 
so in FIG 1) to produce the modified color description date (file 88 in FIG. 1). The process in box 154 applies 
each weight to ite corresponding original color description according to the index in the symbolic pixel expres- 
sion and writes a data structure of modified cotor informatton date with corresponding indices. 

The inquiry in box 138 may also permit the user to specify some user threshold to terminate processing. 
This threshold may specify for example, that each evaluated pbcel must be within some small range of distance 
66 from the gamut (a 'close enough" threshold), or that some percentage of pbcels must be within the gamut to 

conclude processing. , ^. j <• 

When the inquiry in box 138 is not satisfied, the symbolic expressions of the picture function, Po. defined 
by the b selected pixels, are differentiated with respect to the weighte to form the Jacobian matrix, in box 140. 
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and inner loop processing begins in box 142. The inner loop processing represents the specific processing 
steps needed for minimizing the goodness function, which in the illustrated embodiment is the squared Eucli- 
dian distance between two pictures in picture space. As noted above, the Jacobian matrix relates movements 
of the pixels to movements of the weights: J^W| = Ap|. The expression for each pbcel In this matrix contains 

5 symbolic expressions Wi for weights on the spectra. 

In box 142, po is now denoted as pi, since minimizing the distance between two pictures may take several 
iterations of the inner loop, and with each iteration Pi changes as weights are adjusted. In box 142, pi, the set 
of the b worst pixels, is evaluated with respect to the current settings of the weights. These evaluated pbcels 
are then mapped to the device gamut in box 144, using gamut projector function, h, which may be any con- 
to ventional gamut mapping method. The mapping to the gamut of picture pi produces a target picture p*. a list 
of In-gamut pixels. Then, also in box 142, the difference Ap between each pixel color in the current picture Pi 
and the target picture p* is computed. 

In boxes 146 and 148, the specific Jacobian matrix for picture pi is constructed and its pseudo-inverse 
computed, using any standard technique, to find the direction in which to move the weights to bring the current 

15 picture closer to the target picture in picture space. In box 1 50, a step size is selected for the weights, according 
to how nonlinear picture function f(W|) is. In the illustrated embodiment, the step size takes on a user-specified 
value m between 0 and 1. If the largest absolute value in the vector of weight changes computed from the 
pseudo-inverse of the Jacobian is greater than m, the step size is set to m; otherwise, the step size is set to 
one (1). The new weights are found from W|+i Wj + kAW|. 

20 In box 152, an inquiry is made as to whether the inner loop processing is completed. In essence the inquiry 
is whether the current picture of the b worst pixels from the outer loop satisfies the goodness function. The 
inquiry is made here because the most recent set of computed changes for the weights provides Information 
as to how the current picture is "converging" on the target picture. The specific inquiry in box 152, then, includes 
two tests. The first is whether the absolute value of the largest distance difference between the current picture 

25 and the target picture is within a user-specified distance. The second is whether the absolute value of the larg- 
est change in the weights computed from the Jacobian pseudo-inverse is smaller than a user-specified quan- 
tity, that is, too minimal to try another iteration of weight changes. If either test is true, processing of the Inner 
loop is terminated, and processing continues in box 132. If either test is not true, however, the distance function 
may be minimized further, and processing is transferred to the top of the inner loop at box 142. 

30 The method of the present Invention may be used for operating a processor-controlled machine, such as 
a general purpose digital computer or a machine having a digital computer as a component part. In addition, 
the present invention may be a machine which includes a processor for executing instructions which accom- 
plish substantially similar results as the method of the present invention. Figure 10 illustrate a machine con- 
figuration for which the present method and machine Inventions are suitable. Figure 1 0 is a simplified functional 

35 block diagram illustrating the components of a processor-controlled machine 200 (hereinafter, "system 200"). 
System 200 may be a standalone system, such as a microcomputer-based graphics workstation, which Is suit- 
able for performing the scene modeling and creation of the scene description, the rendering of the image with 
a symbolic shaded rendering system, the spectral change calculations method 90 of the present invention, 
and display of the in-gamut final color image on output device 49. The components of system 200 include a 

40 processor 204 and at least one memory unit 202. Memory unit 202 includes ROM and RAM in wh ich are stored 
the program instructions for the scene modeling application, the rendering system, spectral change calculator 
90, and the symbolic pixel expression evaluator. Memory 202 also stores data that may be needed during exe- 
cution of the program instructions, including the scene description 22, indexed color description data 84, the 
symbolic pixel expressions 80, and device gamut data 44. 

45 In system 200, processor 204 is connected for access to and in communication with a respective memory 
unit, and processor 204 controls the execution of program instructions and the retrieval of data from the mem- 
ory. Processor 204 may be any appropriate processor and, in use, it controls the transfer of pixel information 
signals representing pixel image data from memory to output circuitry which includes output unit 49. 

Input unit 206 may include any suitable device and associated input circuitry for interacting with a scene 

50 modeling application, a rendering system 25, and spectral change calculator 90. Suitable devices include but 
are not limited to pointing and cursor control devices fortwo- and three-dimensional displays, such as a mouse 
or light pen, alphanumeric input devices such as a keyboard, and touch screen displays. 

The output units of system 200 may be any appropriately connected color device suitable for displaying 
computer generated color image data, such as a cathode ray tube (CRT) raster scan color monitor or a color 

55 liquid crystal display. Other devices suitable as output units in place of display 49 include color reproduction 
devices, such as an electronic or digital printer, xerographic marking engine, platemaking device, or other suit- 
able color rendering device. In addition, the output unit may be a data communications device for connecting 
to a communications line, as shown in FIGS. 11 and 12, whereby data created according to the method of the 
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present invention may be transmitted to other suitably configured systems for further processing or for display 

or reproduction. . 

In FIG. 11. in an alternative embodiment of the present invention, another machine configuration havmg 
two, similarly configured, processor-controlled machines Is illustrated. System 200 Is connected to system 210 
through communications llnl< 209, which may be any conventional network communications connection be- 
tween the two machines. System 200, which may be a graphics worl^station, is operated to execute rendering 
system Instructions 25 to generate symbolic pixel expression file 80 and indexed color description data 84, 
which together represent the data needed for a rendered color image, and which may be stored in memory 
202. Processor 204 controls the communication of symbolic pixel expression file 80 and Indexed color descrip- 
tion data 84 to receiving system 210 through output unit 208 configured as a communications connection. The 
receiving system 210, which may be a networked, microcomputer or workstation, receives symbolic pixel ex- 
pression file 80 and indexed color description data 84 from system 200 through communications link 209. and 
executes spectral change calculation Instructions 90 and symbolic pixel expression evaluator instructions 82 
to produce the In-gamut Image pixel color data 46 needed for displaying the in-gamut rendered color image 

on output display 49 of system 210. 

A second alternative embodiment Is shown in FIG. 12: there is illustrated still another machine configur- 
ation having three processor-controlled machines connected through communications links 209 and 219. This 
implementation is described in detail in Ref. 1 with reference to FIG. 10 thereof. 

Other machine Implementations of the functional steps of the method of the present Invention are also 
possible. For example, in FIG. 12, system 21 0 could also generate the in-gamut image pixel color data 46 using 
symbolic pbcel expression evaluator 82, thereby creating the data which represents the final color image having 
colors which are in the gamut of designated output medium 49 of system 220. System 220 could then receive 
the in-gamut image pixel color data 46 through communications link 219. for display on output medium 49 with- 
out further processing. The dotted line 88 of FIG. 1 represents the combination of spectral change calculator 
process 90 with symbolic pixel expression evaluator process 82 implemented on a single machine. 

The method of the present Invention was Implemented on a Sun Microsystems SPARCstatlon Model 2 with 
a 24 bit color display, running the Unix BSD 4.2 operating system. The renderer used In the illustrated em- 
bodiment was implemented as a ray tracing rendering method from a simple public domain ray tracer. avaOable 
at the time of Implementation via anonymous ftp from "princeton.edu" under the name of "MTV*. 

The gamut projector function h may be implemented as any suitable gamut mapping process known in the 
art In the Illustrated embodiment, a very simple gamut mapping process is used which projects each out-of- 
gamut pixel in the current ideal image, picture po. onto the position in the gamut in RGB color space closest 
to the out-of-gamut pixel position. Table 2 of Ref. 1 contains the program code, in Mathematica, to implement 
this simple gamut projector function to find the signed distance from an ideal Image pixel color to the closest 

point In the RGB color space: , ^ . * ^ * 

FIGS. 12Aand 12B of Ref. 1 contain an example of the program code, in Mathematica, designated as ref- 
erence numeral 160 in both figures, to implement the steps shown in FIG. 9. 

The present invention furthermore encompasses variations of the above described embodiments which 
will be apparent to persons skilled In the art and. for example, as described on pp. 55-57 of Ref. 1. 



Claims 



1 A method of rendering a colour image from symbolic image data, comprising: 

performing spectral change calculations with symbolic color image data, indexed color description 
data indexed to the symbolic color image data, and device gamut color data representing a gamut of device 
colore of a color reproduction device to produce modified color description data; the modified color de- 
scription data, when used to evaluate the symbolic color Image data, generating a color Image composed 
of a plurality of in-gamut colore; each In-gamut color being one of the device colore in the gamut of the 
color reproduction device. 

2. The method of claim 1 wherein performing spectral change calculations includes the steps of: 

(a) determining color modification data to be used to modify the indexed color description data; 

(b) evaluating the symbolic color image data with the indexed color description data and the color mod- 
ification data to generate a color image composed of modified colore; 

(c) generating a target color image from the color Image using the device gamut color data; each of a 
plurality of target colors composing the target color image being one of the device colore in the gamut 
of the color reproduction device and corresponding to one of the modified colore of the color image; 
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(d) determining whether the color Image satisfies an image metric measuring a valid color image in 
terms of a relationsNp between the color image and the target color image; 

(e) if the color image does not satisfy the Image metric, repeating steps (a), (b), (c), and (d) until the 
color Image satisfies the image metric; and 

(f) modifying the indexed color description data with the color modification data to produce the modified 
color description data. 

The method of claim 2 wherein the image metric measures the valid color image according to a distance 
in a color space between one of the modified colors In the color Image and the corresponding target color 
In the target color image. 

A method of rendering a colour Image from symbolic image data, comprising: 

(a) assigning change variables to symbolic color Image data items; 

(b) evaluating the symbolic color image data items with indexed color description data items to generate 
a current color Image composed of current colors; 

(c) generating, from the current colors of the current color Image, a target color image using device 
gamut color data; 

(d) calculating difference data between the current colors and the target colors; 

(e) determining incremental change quantities for the values of the change variables using the differ- 
ence data; 

(f) determining from the difference data and from the incremental change quantities whether the cur- 
rent color image satisfies Image metric data; the image metric data including relationship data for meas- 
uring a valid color image in terms of a relationship between the current color image and the target color 
image, and change variable data for measuring a minimum acceptable quantity for the incremental 
change quantities; 

(g) if the current color image does not satisfy the image metric data, updating the current values of 
the change variables with the incremental change quantities, and repeating steps (b), (c), (d). (e), (f) 
and (g); and 

(h) applying the current values of the change variables to the indexed color description data to produce 
modified color description data. 

The method of claim 4 wherein the relationship data of the image metric data measures the current color 
image according to a distance In a color space between one of the current colors In the cunrent color image 
and the corresponding target color in the target color image. 

A method of operating a machine for rendering a colour image from symbolic image data, the machine 
including: 

a memory for storing data; and 

a processor connected for accessing and storing data In the memory; 
the data stored in the memory including 

indexed color description data items; each indexed color description data item defining an original 
color of one of a plurality of object primitives in a scene description; 

symbolic color Image data items having symbolic spectral components indexing the Indexed color 
description data items to the symbolic color image data items; each symbolic color image data item de- 
fining one of a plurality of image pixel color data items composing a color image rendered from the scene 
description; 

device gamut color data Items representing for a color reproduction device a gamut of device colors 
capable of reproduction by the device; and 

image metric data representing a desired relationship between the plurality of image pixel color 
data items composing the color image and a plurality of target image pixel color data items composing a 
target color image; 

the method comprising the following steps carried out by the processor 

(a) assigning change variables to the symbolic spectral components in the symbolic color image data 
items stored in the memory of the machine; 

(b) determining a new value for at least one change varia bte to produce modified symbolic spectral com- 
ponents; 

(c) evaluating each modified symbolic spectral component of each symbolic color image data Item with 
the indexed color description data items to produce the Image pixel color data items composing the 
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color image; 

(d) generating the plurality of target Image pixel color data Items composing the target color image from 
the Image pixel color data items composing the color image using the device gamut color data Items; 
each target Image pixel color data item representing one of the device colors capable of reproduction 
by the device; 

(e) determining If the color Image satisfies the relationship represented by the image metric data; 

(f) If the color Image does not satisfy the Image metric data, repeating steps (b), (c). (d), and (e) until 
the color Image satisfies the image metric data; and 

(g) applying the new value of each change variable to the indexed color description data item indexed 
to the symbolic spectral component to generate a plurality of modified Indexed color description data 
items. 

A machine for rendering a colour image from symbolic Image data, comprising: 

Input circuitry for obtaining symbolic color Image data items and indexed color description data 
items; each Indexed color description data item defining an original color of one of a plurality of object 
primitives in a scene description; each symbolic color image data Item defining one of a plurality of Image 
pbcel color data Items composing a color Image rendered from the scene description; the symbolic color 
image data Items having symbolic spectral components indexing the Indexed color description data Items 
to the symbolic color Image data items; 
memory for storing data; 

output circuitry for connecting the machine to an output medium and for transferring selected data 

from the memory to the output medium; and 

a processor connected for accessing the input circuitry to receive the symbolic color image data 
Items and the indexed color description data items and to store the symbolic color image data items and 
the indexed color description data items in the memory; the processor further being connected for ac- 
cessing the output circuitry to transfer the selected data from the memory to the output circuitry; 

the processor further being connected for accessing data stored in the memory; the data stored 

in the memory including: . ^. u 

device gamut color data items representing a gamut of device colors capable of reproduction by a 

color reproduction device; and 

instruction data indicating instructions the processor executes; 

the processor, In executing the Instructions, performs spectra! change calculations with the sym- 
bolic color Image data items, the indexed color description data Items, and the device gamut color data 
Items to produce modified color description data items capable of producing, together with the symbolic 
color image data items, the plurality of image pixel color data Items composing the color Image; each im- 
age pixel color data Item representing one of the device colors In the gamut of the color reproduction de- 
vice; the processorfurther. in executing the Instructions, stores the modified color description data items 
in the memory. 

The machine of daim 8 wherein the processorfurther, in executing the instructions, (1) selects the modi- 
fied color description data Items and the symbolic color Image data items in the memory and transfers 
the selected data Items from the memory to the output circuitry; the output circuitry transferring the modi- 
fied color description data items and the symbolic color image data Items to the connected output medium 
or (2) evaluates the symbolic color image data items with the modified color description data Items to gen- 
erate the image pixel color data items composing the color image; and wherein the processor further, in 
executing the Instructions, transfers the plurality of image pixel color data items to the output circuitry for 
transferring to the output medium. 

The machine of claim 8 or 9 wherein the output medium is (1) the color reproduction device having the 
gamut of device colors represented by the device gamut color data items stored in the memory of the 
machine or (2) a data communications circuit 
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